add ca fertilization harmonization and ncc compost support#4002
add ca fertilization harmonization and ncc compost support#4002divine7022 wants to merge 100 commits into
Conversation
|
high level, priors are out now, two ref tables (ca_compost_amendment, ca_n_application_rate) stay as bundled data for now. and new ag/management package vs bety priors vs whatever is separate conv |
@divine7022 Could you please clarify? |
dlebauer
left a comment
There was a problem hiding this comment.
Thank you for these changes.
General changes requested:
Better documentation
(Make sure to review the data packaging guidance from the Data chapter of the "R-Packages" book)
Documentation should cover both the source (TSV) and package (RDA) data:
package datasets should be documented in roxygen, and should include
- definitions of each field
- information about where the data came from:
- references
- how the dataset was compiled, including assumptions
- link to more details in source data README.
Source datasets can be documented in a data-raw/README.md and should include:
- where data came from
- what decisions were made during curation
- note that for package datasets that do not have a manual curation step, the provenance of package data is documented by R scripts.
- how to update both the TSV and .rda files
dataset names
Make sure naming is consistent, e.g. create_n_rate_data.R reads a file n_fertilizaton.tsv and generates a dataset ca_n_application_rate. This can get confusing.
consider renaming compost.tsv to organic_ammendments.tsv to bettr reflect the diversity of materials represented in the dataset.
| @@ -0,0 +1,33 @@ | |||
| Material C_MIN (C:N) C_MAX (C:N) C_Avg (C:N) C_Assumed (%) Total N (%) 4 week PAN (%) LowerN/HigherN RowsMIN_AppRate (tons/acre) RowsMIN_AppRate (lbs/acre) RowsMIN_Total_N (lbs N/acre) RowsMIN_Avail_N (lbs N/acre) RowsMIN_Total_C (lbs C/acre) RowsMAX_AppRate (tons/acre) RowsMAX_AppRate (lbs/acre) RowsMAX_Total_N (lbs N/acre) RowsMAX_Avail_N (lbs N/acre) RowsMAX_Total_C (lbs C/acre) TreesMIN_AppRate (tons/acre) TreesMIN_AppRate (lbs/acre) TreesMIN_Total_N (lbs N/acre) TreesMIN_Avail_N (lbs N/acre) TreesMIN_Total_C (lbs C/acre) TreesMAX_AppRate (tons/acre) TreesMAX_AppRate (lbs/acre) TreesMAX_Total_N (lbs N/acre) TreesMAX_Avail_N (lbs N/acre) TreesMAX_Total_C (lbs C/acre) Source | |||
There was a problem hiding this comment.
This table combines stoichiometric properties with assumed application-rate
scenarios.
- would it make more sense to separate material properties from application-rates?
- See more general requests about dataset documentation in my general review comments. Some specific questions are below
- what columns come from the source, and what information was added during manual curation?
- e.g. are the
Rows*andTrees*application-rate values directly supported by
the cited sources, or did sources provide more granular, species level rates?
- e.g. are the
- Why does the build script expose only the
Rows*values in
ca_compost_amendment? - Is the intended distinction that synthetic N rates are crop-specific, while
compost/organic amendment rates are only available at a broader crop-structure
level? - Given the contents include manure, blood meal, paper, wood chips, straw, and
other organic materials, wouldorganic_amendments.tsvor
compost_amendments.tsvbe a clearer filename thancompost.tsv?
There was a problem hiding this comment.
yeh did the split. one table for the properties (ca_organic_amendment_properties, so C:N, total N, 4-week PAN, material_class, source), one for the rates (ca_organic_amendment_app_rate, long by material and crop_structure). they join on material
to your other questions:
what columns come from the source, and what information was added during manual curation?
pretty much all of it's straight off the sheet, except material_class (we set that by hand, more on it below), the g/m2 columns (converted from lbs/acre with ud_convert), and the short citations (we mapped those from the source urls). it's laid out in the readme.
are the Rows* and Trees* application-rate values directly supported by
the cited sources, or did sources provide more granular, species level rates?
sheet has both. for 30 of the 32 it's the same number copied across, but 2 of em (blood meal, fresh poultry manure) really do differ and trees comes in lower. the sources only go down to crop-structure, not species, so that's as fine as it gets.
Why does the build script expose only the Rows* values..
yeh that's flagged, old script dropped Trees. both are in now.
Is the intended distinction that synthetic N rates are crop-specific, while
compost/organic amendment rates are only available at a broader crop-structure
level?
ryt, exactly. the N rates are per crop, but amendment rates only come per crop structure (rows = annual, trees = orchard), so the two tables get keyed differently on purpose
| # map each raw material name to one of the CalRecycle classes | ||
| # (14 CCR section 17852). biosolids is empty in the current table. | ||
| material_to_class <- function(m) { | ||
| s <- tolower(m) |
There was a problem hiding this comment.
where does m come from? I don't see it in compost.tsv
There was a problem hiding this comment.
yeh good catch, that whole function's gone. back when the harmonizing was in the build script, material_to_class() was working out the class on the fly with grepl. m was just the material name getting passed in, but yeh, you'd have to go chase the call to see that, which is the whole problem. now it's a plain column in the source data, one class per material, mapped to CalRecycle list (14 CCR 17852). just data you can read instead of code you have to trace
| app_rate_min = .data$`RowsMIN_AppRate (lbs/acre)`, | ||
| app_rate_max = .data$`RowsMAX_AppRate (lbs/acre)`, |
There was a problem hiding this comment.
why are only Rows[MIN|MAX] and not Trees[MIN|MAX] used here? (see also general question about documentation)
There was a problem hiding this comment.
not on purpose. same deal as the m thing, when the harmonizing was jammed into the build script it only wired up the Rows columns and quietly dropped Trees. now the harmonize step is separate script on scc and it splits Rows/Trees into the long table by crop_structure, so both stick around. most materials have the same rate either way, but 2 that don't (blood meal, fresh poultry manure) keep their tree rate now instead of losing it
| @@ -0,0 +1,90 @@ | |||
| PFT Group Crop PlantStage Season MINN MAXN Unit Source Notes | |||
There was a problem hiding this comment.
Do crop-names come from a controlled vocabulary? Any reason not to use the LandIQ names here?
Where do PFTs come from? On one hand, row/woody is a simple designation, and there may be reasons to include a grouping in this dataset. But caution should be used and rationale/motivation documented because PFT groupings are opinionated, and in this case alfalfa is not typically considered a 'row' crop (it is forage).
There was a problem hiding this comment.
two things here
crop names: yeh those are FREP / UC ANR names off the extension pubs, not LandIQ, and that's on purpose. the rates get published at that level, and some landIQ subclasses (like "miscellaneous deciduous") you can't really pin one rate to. landIQ to FREP mapping lives over in the workflow's Crop_types crosswalk, not in this table
pft: fair, and yeh ryt, alfalfa's forage not row crop. honestly pft_group is just hand set in the spreedsheet, it's a rough row/woody/rice bucket that lines up with grass vs deciduous pfts, not a real botanical/ag call. I kept it matching the canonical cadwr_pfts.csv, which also calls alfalfa row (it's under the pasture class there). I did try pulling pft_group straight from cadwr during harmonize so it'd be one source of truth, but it came out worse, the crosswalk sends oats into the misc grain and hay bucket so it flips to "hay", and about 9 crops don't match cleanly and just drop out. so it stays hand set, and I wrote the reasoning plus the alfalfa caveat into the docs
There was a problem hiding this comment.
To unblock #4003, which currently depends on this PR, you could
- copy the .rda and .R files from this PR to the
workflows/[fertilization|ncc]-statewidedirectories - update #4003 to use those resources
This is in line with one of the motivations for the workflows directory - as a place to maintain and develop workflows until individual components can be generalized and extracted into PEcAn package functions and data
|
quick context first, since this turned into a structural change. where we started, the build scripts here were reading the curated fertilization data straight from Mike's team's spreedsheet, and doing the harmonization in place mapping the source URLs to citations, the unit conversions, the property vs rate columns, the material classes. So the .rda we shipped was harmonized version, but all that harmonization logic was baked into package build per discussion, now I flipped it. harmonization moved out to a standalone harmonize_fertilization_data.R on SCC. It reads original TSVs and writes already harmonized CSVs, those CSVs land in data-raw/, and create_*.R scripts just read them and call use_data(). No curation left in the package, it's purely packaging clean data now. That's the "no manual curation step, provenance is the R script" case you pointed at regarding docs, rewrote the roxygen for all three datasets, and added a data-raw/README.md covering where the raw sheet comes from, the curation decisions, and how to update
corrected, and went past the rename. now names lines up: ca_n_application_rate.csv, create_ca_n_application_rate.R, ca_n_application_rate, same for amendments. I have dropped compost.tsv, instead of just renaming it I split it into ca_organic_amendment_properties and ca_organic_amendment_app_rate (more in ^^ thread) and dropped "compost" name since the table's mostly manure, straw, wood chips, paper, blood meal |
Description
folds standalone N fertilization harmonization (was living at /projectnb/dietzelab/ccmmf/management/fertilization/harmonization.R) into data.land data-raw pipeline, then layers ncc (compost) sampling support on top and extends the events schema with an ncc event type
events_schema_v0.1.2.json picks up ncc as an allowed event_type with material, ncc_subtype, fert_subtype, and pft properties
separate workflow PR will consume these samplers from workflows/fertilization-statewide and workflows/ncc-statewide to emit ensemble events
cc @sarahkanee @mdietze @infotroph @dlebauer
Motivation and Context
Review Time Estimate
Types of changes
Checklist: